ci: verify ecp-elastic-agent-service image is multiarch #11498

rubenruizdegauna · 2025-12-01T09:35:55Z

This pull request adds a new validation step to the Buildkite pipeline to ensure that the agentless Docker image is built for all required architectures before running service tests. The main change is the introduction of a script that checks the presence of both amd64 and arm64 architectures in the Docker image manifest for the specified environment.

Pipeline enhancements:

Added a new pipeline step labeled ":docker: Validate docker image is built for all architectures" in .buildkite/pipeline.agentless-tests.yaml to run the validation script before service tests.

Docker image validation:

Introduced .buildkite/scripts/steps/validate-agentless-docker-image.sh, a script that:
- Extracts the relevant image SHA from the environment configuration.
- Inspects the Docker image manifest to determine available architectures.
- Validates that both amd64 and arm64 architectures are present, failing the build if any are missing.

mergify · 2025-12-01T09:36:40Z

This pull request does not have a backport label. Could you fix it @rubenruizdegauna? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
backport-active-all is the label that automatically backports to all active branches.
backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

elasticmachine · 2025-12-01T11:41:53Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

v1v

LGTM, just some minor comments with using container agents for faster builds and CODEOWNERS to help with notifying the corresponding team, instead the robots team

v1v · 2025-12-01T13:10:31Z

.buildkite/pipeline.agentless-tests.yaml

+  - label: ":docker: Validate docker image is built for all architectures"
+    command: ".buildkite/scripts/steps/validate-agentless-docker-image.sh"
+    agents:
+      provider: "gcp"


nit:

IIUC, we use this provider since it requires using Docker in it. Wonder whether we could use a container agent here with docker on it? like https://hub.docker.com/_/docker

I have not tested btw

I thought about it, but I wasn't sure which one to use.

My first approach was to try docker.elastic.co/ci-agent-images/quality-gate-seedling and docker.elastic.co/ci-agent-images/platform-ingest/buildkite-agent-beats-ci-with-hooks:0.5 but none of the have docker.

Then reading the docs I found this

https://docs.elastic.dev/ci/getting-started-with-buildkite-at-elastic#run-docker-in-a-build

Since container management is not supported in the Kubernetes offering, in order to use docker-like commands, you need to use a GCP virtual machine agent.

So I understood that this was the correct way.

Is it ok to use image that are not from docker.elastic.co?

If that's the case, I can give it a try to https://hub.docker.com/_/docker

I would strongly prefer if we used something from docker.elastic.co (reasons also include failure domains etc). I have yet to find any image of ours that have docker-in-docker in it.

In this case though, we don't technically need docker, but any of the manifest tooling could achieve this, e.g using oras (going from bash/oras/yq to pseudo-code so bear with me):

$ image_archs=$(oras manifest fetch docker.elastic.co/observability-ci/ecp-elastic-agent-service:git-b1d4e25c181a | yq '[.manifests[].platform.architecture] | sort | join(", ")') $ echo $image_archs amd64, arm64 # assert image_archs == amd64, arm64

Modified to use custom image docker.elastic.co/ci-agent-images/observability/oci-image-tools-agent

(created in https://github.com/elastic/ci-agent-images/pull/2052)

v1v · 2025-12-01T13:10:57Z

.buildkite/scripts/steps/validate-agentless-docker-image.sh

@@ -0,0 +1,111 @@
+#!/usr/bin/env bash


shall we add an entry in the CODEOWNERS with the team responsible for this script and probably .buildkite/pipeline.agentless-tests.yaml?

Good point. Added in 36c8137

v1v · 2025-12-01T13:17:01Z

.buildkite/scripts/steps/validate-agentless-docker-image.sh

+setup_extract_sha() {
+    # Ensure repo is available - redirect output to /dev/null
+    if [ ! -d "serverless-gitops" ]; then
+        git clone --depth 1 [email protected]:elastic/serverless-gitops.git


nit: can you confirm whether this is accessible in BK? I cannot find the relevant GH check for testing this PR Maybe I'm blind. Do you mind pointing where I can see it? 🙇

This was inspired by https://github.com/elastic/elastic-agent/blob/main/.buildkite/scripts/steps/run-agentless-tests.sh#L26 which is triggered from the same bk job, so permissions should be correct.

for testing this PR
TBH I haven't tested this in the pipeline. Just locally assuming that the permissions will work as the same checkout is done in the agentless-tests.

But I can create a branch instead of a fork and trigger a test :)

That's ok, I assume both scripts are triggered in the same bk pipeline.

I somehow thought it runs automatically for forked PRs too

It was a good idea in any case, as I found a couple of issues (yq not present and missing wait between steps), so thanks for the ping ;-)

ycombinator

Approving to unblock PR from merging (needs approval from @elastic/elastic-agent-control-plane until the CODEOWNERS changes in this PR take effect) but will defer to someone from @elastic/ingest-managed-jobs for a second set of eyes.

swiatekm

LGTM. I'm not overly enthused about this being a bash script, but tying it into agent build tooling would probably be worse in the long run.

nkvoll · 2025-12-02T13:18:44Z

.buildkite/scripts/steps/validate-agentless-docker-image.sh

+setup_extract_sha
+
+# Extract the SHA for the specified environment
+COMMIT_SHA=$(extract_sha "$ENVIRONMENT")


I think we can replace everything up to this point with ${SERVICE_VERSION}?

Good point. I checked the BK env vars and SERVICE_VERSION is there so I removed the code that was retrieving it.

nkvoll · 2025-12-02T13:31:41Z

.buildkite/pipeline.agentless-tests.yaml

+  - label: ":docker: Validate docker image is built for all architectures"
+    command: ".buildkite/scripts/steps/validate-agentless-docker-image.sh"
+    agents:
+      provider: "gcp"


I would strongly prefer if we used something from docker.elastic.co (reasons also include failure domains etc). I have yet to find any image of ours that have docker-in-docker in it.

In this case though, we don't technically need docker, but any of the manifest tooling could achieve this, e.g using oras (going from bash/oras/yq to pseudo-code so bear with me):

$ image_archs=$(oras manifest fetch docker.elastic.co/observability-ci/ecp-elastic-agent-service:git-b1d4e25c181a | yq '[.manifests[].platform.architecture] | sort | join(", ")') $ echo $image_archs amd64, arm64 # assert image_archs == amd64, arm64

elasticmachine · 2025-12-03T18:27:17Z

💚 Build Succeeded

Buildkite Build
Commit: cebd2b6

History

💛 Build #31228 was flaky 01e00b5
💚 Build #31208 succeeded 6ae509c

cc @rubenruizdegauna

ci: verify ecp-elastic-agent-service image is multiarch

6ae509c

rubenruizdegauna requested a review from a team December 1, 2025 09:35

rubenruizdegauna requested review from a team as code owners December 1, 2025 09:35

rubenruizdegauna requested review from swiatekm and ycombinator December 1, 2025 09:35

mergify bot assigned rubenruizdegauna Dec 1, 2025

rubenruizdegauna added skip-changelog backport-skip labels Dec 1, 2025

pierrehilbert added the Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team label Dec 1, 2025

v1v previously approved these changes Dec 1, 2025

View reviewed changes

v1v reviewed Dec 1, 2025

View reviewed changes

add agentless bk files to CODEOWNERS

36c8137

rubenruizdegauna dismissed v1v’s stale review via 36c8137 December 1, 2025 13:32

rubenruizdegauna added 3 commits December 1, 2025 14:40

fix typo in codeowners

a7371a6

do not start the agentless-tests if the 1st step fails

f83789a

install yq

01e00b5

rubenruizdegauna requested a review from v1v December 1, 2025 14:55

ycombinator previously approved these changes Dec 1, 2025

View reviewed changes

swiatekm previously approved these changes Dec 2, 2025

View reviewed changes

nkvoll reviewed Dec 2, 2025

View reviewed changes

use custom image. simplify script

f6432ab

rubenruizdegauna dismissed stale reviews from swiatekm and ycombinator via f6432ab December 3, 2025 16:26

rubenruizdegauna added 2 commits December 3, 2025 17:26

disable test running just to test the 1st step

2072b50

fix agent

cebd2b6

rubenruizdegauna marked this pull request as draft December 3, 2025 17:22

ci: verify ecp-elastic-agent-service image is multiarch #11498

Are you sure you want to change the base?

ci: verify ecp-elastic-agent-service image is multiarch #11498

Conversation

rubenruizdegauna commented Dec 1, 2025

Uh oh!

mergify bot commented Dec 1, 2025

Uh oh!

elasticmachine commented Dec 1, 2025

Uh oh!

v1v left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ycombinator left a comment

Choose a reason for hiding this comment

Uh oh!

swiatekm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Dec 3, 2025

💚 Build Succeeded

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants